Solving POMDPs by Searching in Policy Space

Most algorithms for solving POMDPs itera­ tively improve a value function that implic­ itly represents a policy and are said to search in value function space. This paper presents an approach to solving POMDPs that repre­ sents a policy explicitly as a finite-state con­ troller and iteratively improves the controller by search in policy space. Two related al­ gorithms illustrate this approach. The first is a policy iteration algorithm that can out­ perform value iteration in solving infinite­ horizon POMDPs. It provides the founda­ tion for a new heuristic search algorithm that promises further speedup by focusing compu­ tational effort on regions of the problem space that are reachable, or likely to be reached, from a start state.